{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7a5d459f",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/uptrain-ai/uptrain/blob/main/examples/checks/conversation/guideline_adherence.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f74cbf3-07b5-40d8-86cd-6fe807d8f36e",
   "metadata": {
    "tags": []
   },
   "source": [
    "<h1 align=\"center\">\n",
    "  <a href=\"https://uptrain.ai\">\n",
    "    <img width=\"300\" src=\"https://user-images.githubusercontent.com/108270398/214240695-4f958b76-c993-4ddd-8de6-8668f4d0da84.png\" alt=\"uptrain\">\n",
    "  </a>\n",
    "</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fc0b549-d081-442e-87ce-1ed0657540f2",
   "metadata": {},
   "source": [
    "<h1 style=\"text-align: center;\">Evaluating Guideline Adherence in Conversations</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f92208d-e85c-4108-951d-4f54940ace43",
   "metadata": {
    "tags": []
   },
   "source": [
    "**What is Guideline Adherence?**: Guideline adherence refers to the extent to which the LLM follows a given guideline, rule, or protocol. Given the complexity of LLMs, it is crucial to define certain guidelines, be it in terms of the structure of the output or the constraints on the content of the output or protocols on the decision-making capabilities of the LLM (agents). \n",
    "\n",
    "For example, for an LLM-powered chatbot agent trained to perform appointment booking tasks only, you want to make sure that the LLM is following the guideline: \"The agent should redirect all the queries to the human agent, except the ones related to appointment booking.\"\n",
    "\n",
    "**Data schema**: The data schema required for this evaluation is as follows:\n",
    "\n",
    "| Column Name | Description |\n",
    "| ----------- | ----------- |\n",
    "| conversation | The conversation between the user and the LLM |\n",
    "| guideline | Explanation of the guideline to be followed |\n",
    "| guideline_name | Naming the guideline for better accessibility |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82c5b58f-bfb9-4c63-8b22-8ed05fa1fa4d",
   "metadata": {},
   "source": [
    " If you face any difficulties, need some help with using UpTrain or want to brainstorm on custom evaluations for your use-case, [speak to the maintainers of UpTrain here](https://calendly.com/uptrain-sourabh/30min).\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f936a36",
   "metadata": {},
   "source": [
    "## Step 1: Install UpTrain by running 'pip install uptrain'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "37a5ab48",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: uptrain in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (0.7.0)\n",
      "Requirement already satisfied: pydantic in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2.6.4)\n",
      "Requirement already satisfied: pydantic-settings in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2.2.1)\n",
      "Requirement already satisfied: loguru in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.7.0)\n",
      "Requirement already satisfied: lazy-loader in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.3)\n",
      "Requirement already satisfied: networkx in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (3.1)\n",
      "Requirement already satisfied: polars>=0.18 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.20.13)\n",
      "Requirement already satisfied: pandas in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2.0.3)\n",
      "Requirement already satisfied: numpy>=1.23.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.26.3)\n",
      "Requirement already satisfied: httpx>=0.24.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.26.0)\n",
      "Requirement already satisfied: plotly>=5.0.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (5.15.0)\n",
      "Requirement already satisfied: aiolimiter>=1.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.1.0)\n",
      "Requirement already satisfied: openai>=1.6.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.6.1)\n",
      "Requirement already satisfied: fsspec in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2023.6.0)\n",
      "Requirement already satisfied: litellm in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.34.25)\n",
      "Requirement already satisfied: pyyaml in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (6.0.1)\n",
      "Requirement already satisfied: json5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.9.24)\n",
      "Requirement already satisfied: python-docx in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.1.0)\n",
      "Requirement already satisfied: pymupdf in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.23.25)\n",
      "Requirement already satisfied: faiss-cpu in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.7.4)\n",
      "Requirement already satisfied: anyio in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (3.7.1)\n",
      "Requirement already satisfied: certifi in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (2023.7.22)\n",
      "Requirement already satisfied: httpcore==1.* in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (1.0.2)\n",
      "Requirement already satisfied: idna in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (3.4)\n",
      "Requirement already satisfied: sniffio in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (1.3.0)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpcore==1.*->httpx>=0.24.1->uptrain) (0.14.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from openai>=1.6.1->uptrain) (1.8.0)\n",
      "Requirement already satisfied: tqdm>4 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from openai>=1.6.1->uptrain) (4.66.2)\n",
      "Requirement already satisfied: typing-extensions<5,>=4.7 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from openai>=1.6.1->uptrain) (4.7.1)\n",
      "Requirement already satisfied: tenacity>=6.2.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from plotly>=5.0.0->uptrain) (8.2.3)\n",
      "Requirement already satisfied: packaging in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from plotly>=5.0.0->uptrain) (23.2)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pydantic->uptrain) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.16.3 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pydantic->uptrain) (2.16.3)\n",
      "Requirement already satisfied: aiohttp in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (3.9.1)\n",
      "Requirement already satisfied: click in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (8.1.6)\n",
      "Requirement already satisfied: importlib-metadata>=6.8.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (6.8.0)\n",
      "Requirement already satisfied: jinja2<4.0.0,>=3.1.2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (3.1.2)\n",
      "Requirement already satisfied: python-dotenv>=0.2.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (1.0.0)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.31.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (2.31.0)\n",
      "Requirement already satisfied: tiktoken>=0.4.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (0.4.0)\n",
      "Requirement already satisfied: tokenizers in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (0.15.1)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pandas->uptrain) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pandas->uptrain) (2023.3)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pandas->uptrain) (2023.3)\n",
      "Requirement already satisfied: PyMuPDFb==1.23.22 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pymupdf->uptrain) (1.23.22)\n",
      "Requirement already satisfied: lxml>=3.1.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from python-docx->uptrain) (4.9.3)\n",
      "Requirement already satisfied: zipp>=0.5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from importlib-metadata>=6.8.0->litellm->uptrain) (3.16.2)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from jinja2<4.0.0,>=3.1.2->litellm->uptrain) (2.1.3)\n",
      "Requirement already satisfied: six>=1.5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->uptrain) (1.16.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.31.0->litellm->uptrain) (3.2.0)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.31.0->litellm->uptrain) (1.26.16)\n",
      "Requirement already satisfied: regex>=2022.1.18 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from tiktoken>=0.4.0->litellm->uptrain) (2023.8.8)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (23.1.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.9.2)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.4.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.3.1)\n",
      "Requirement already satisfied: huggingface_hub<1.0,>=0.16.4 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from tokenizers->litellm->uptrain) (0.20.3)\n",
      "Requirement already satisfied: filelock in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers->litellm->uptrain) (3.12.2)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install uptrain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb91700c",
   "metadata": {},
   "source": [
    "## Step 2: Let's define our dataset to run evaluations upon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bf4c2b81-59f4-4faa-b474-7b3f4c2ed509",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "satisfactory_chat = [{\n",
    "    'conversation' : [\n",
    "        {\"role\": \"patient\", \"content\": \"Help\"}, \n",
    "        {\"role\": \"nurse\", \"content\": \"what do you need\"}, \n",
    "        {\"role\": \"patient\", \"content\": \"Having chest pain\"}, \n",
    "        {\"role\": \"nurse\", \"content\": \"please call 102\"},\n",
    "        {\"role\": \"patient\", \"content\": \"Thank you nurse\"}, \n",
    "    ]  \n",
    "}]\n",
    "\n",
    "unsatisfactory_chat = [{\n",
    "    'conversation' : [\n",
    "        {\"role\": \"patient\", \"content\": \"Help\"}, \n",
    "        {\"role\": \"nurse\", \"content\": \"what do you need\"}, \n",
    "        {\"role\": \"patient\", \"content\": \"Having chest pain\"}, \n",
    "        {\"role\": \"nurse\", \"content\": \"Sorry, I am not sure what that means\"},\n",
    "        {\"role\": \"patient\", \"content\": \"You don't understand. Do something! I am having severe pain in my chest\"}\n",
    "    ]  \n",
    "}]\n",
    "\n",
    "data = satisfactory_chat + unsatisfactory_chat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b640784e",
   "metadata": {},
   "source": [
    "## Step 3: Running evaluations using UpTrain's Open-Source Software (OSS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d6706deb-8e9d-4d45-8f9b-aed0e9da608f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages/lazy_loader/__init__.py:185: RuntimeWarning: subpackages can technically be lazily loaded, but it causes the package to be eagerly loaded even if it is already lazily loaded.So, you probably shouldn't use subpackages with this lazy feature.\n",
      "  warnings.warn(msg, RuntimeWarning)\n",
      "\u001b[32m2024-05-16 14:44:30.379\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36muptrain.framework.evalllm\u001b[0m:\u001b[36mevaluate_on_server\u001b[0m:\u001b[36m387\u001b[0m - \u001b[1mSending evaluation request for rows 0 to <50 to the Uptrain\u001b[0m\n",
      "\u001b[32m2024-05-16 14:44:43.713\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36muptrain.framework.evalllm\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m376\u001b[0m - \u001b[1mLocal server not running, start the server to log data and visualize in the dashboard!\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from uptrain import EvalLLM, ConversationGuidelineAdherence\n",
    "import json\n",
    "\n",
    "OPENAI_API_KEY = \"sk-******************************\"  # Insert your OpenAI key here\n",
    "\n",
    "eval_llm = EvalLLM(openai_api_key=OPENAI_API_KEY)\n",
    "\n",
    "res = eval_llm.evaluate(\n",
    "    data = data,\n",
    "    checks = [ConversationGuidelineAdherence(guideline=\"Provide emergency contact information if the patient is in distress\", guideline_name=\"Emergency Contact Information\")],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fa5df117-5516-4e32-b86b-e3ade08a5d78",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Help\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"what do you need\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Having chest pain\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"please call 102\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Thank you nurse\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_conversation_Emergency Contact Information_adherence\": 1.0,\n",
      "      \"explanation_conversation_Emergency Contact Information_adherence\": \"[\\\"The given conversation strictly adheres to the guideline of providing emergency contact information if the patient is in distress. In the conversation, the patient expresses 'Help' indicating distress, followed by mentioning 'Having chest pain'. The nurse immediately responds by instructing to 'please call 102', which is a clear indication of providing emergency contact information. The conversation ends with the patient thanking the nurse, showing that the guideline was followed by providing the necessary information in case of distress.\\\"]\"\n",
      "   },\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Help\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"what do you need\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Having chest pain\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Sorry, I am not sure what that means\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"You don't understand. Do something! I am having severe pain in my chest\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_conversation_Emergency Contact Information_adherence\": 0.0,\n",
      "      \"explanation_conversation_Emergency Contact Information_adherence\": \"[\\\"The given conversation strictly violates the guideline of providing emergency contact information if the patient is in distress. Despite the patient clearly expressing distress by mentioning 'Having chest pain' and emphasizing the severity of the situation with 'I am having severe pain in my chest', the nurse fails to provide any emergency contact information. Instead, the nurse responds with uncertainty and does not take appropriate action to address the patient's distress. This violation of the guideline puts the patient at risk by not providing the necessary support in a critical situation.\\\"]\"\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res,indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49ea84a4-2846-4a3c-97c2-ac8c8d413c52",
   "metadata": {},
   "source": [
    "## Step 4: Let's look at some of the results "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db249e6d-5a30-43aa-92f0-6217f6ca85f3",
   "metadata": {},
   "source": [
    "### Sample with a good conversation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6343e0c0-35a1-4d59-a771-dd8a0116fb50",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "   \"conversation\": [\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Help\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"what do you need\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Having chest pain\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"please call 102\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Thank you nurse\"\n",
      "      }\n",
      "   ],\n",
      "   \"score_conversation_Emergency Contact Information_adherence\": 1.0,\n",
      "   \"explanation_conversation_Emergency Contact Information_adherence\": \"[\\\"The given conversation strictly adheres to the guideline of providing emergency contact information if the patient is in distress. In the conversation, the patient expresses 'Help' indicating distress, followed by mentioning 'Having chest pain'. The nurse immediately responds by instructing to 'please call 102', which is a clear indication of providing emergency contact information. The conversation ends with the patient thanking the nurse, showing that the guideline was followed by providing the necessary information in case of distress.\\\"]\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res[0],indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a7388f1",
   "metadata": {},
   "source": [
    "### Sample with a bad conversation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9c1a6497",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "   \"conversation\": [\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Help\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"what do you need\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Having chest pain\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"Sorry, I am not sure what that means\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"You don't understand. Do something! I am having severe pain in my chest\"\n",
      "      }\n",
      "   ],\n",
      "   \"score_conversation_Emergency Contact Information_adherence\": 0.0,\n",
      "   \"explanation_conversation_Emergency Contact Information_adherence\": \"[\\\"The given conversation strictly violates the guideline of providing emergency contact information if the patient is in distress. Despite the patient clearly expressing distress by mentioning 'Having chest pain' and emphasizing the severity of the situation with 'I am having severe pain in my chest', the nurse fails to provide any emergency contact information. Instead, the nurse responds with uncertainty and does not take appropriate action to address the patient's distress. This violation of the guideline puts the patient at risk by not providing the necessary support in a critical situation.\\\"]\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res[1],indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2904e2f4",
   "metadata": {},
   "source": [
    "## [Optional] Step 5: UpTrain Managed Service and Dashboards"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bb801c3",
   "metadata": {},
   "source": [
    "You can create a free UpTrain account [here](https://uptrain.ai/) and get free trial credits. If you want more trial credits, [book a call with the maintainers of UpTrain here](https://calendly.com/uptrain-sourabh/30min)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5e23c4a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-05-16 16:47:32.739\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36muptrain.framework.remote\u001b[0m:\u001b[36mlog_and_evaluate\u001b[0m:\u001b[36m677\u001b[0m - \u001b[1mSending evaluation request for rows 0 to <50 to the Uptrain server\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Help\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"what do you need\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Having chest pain\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"please call 102\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Thank you nurse\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_Emergency Contact Information_adherence\": 1.0,\n",
      "      \"explanation_Emergency Contact Information_adherence\": \"The assistant strictly adhered to the guideline by providing emergency contact information when the patient mentioned having chest pain. The nurse promptly responded by instructing to call 102, which aligns with the guideline of providing emergency contact information in case of distress.\"\n",
      "   },\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Help\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"what do you need\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Having chest pain\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Sorry, I am not sure what that means\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"You don't understand. Do something! I am having severe pain in my chest\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_Emergency Contact Information_adherence\": 0.0,\n",
      "      \"explanation_Emergency Contact Information_adherence\": \"The nurse fails to provide emergency contact information despite the patient expressing distress and severe chest pain. The guideline explicitly states to provide emergency contact information in such situations, which the nurse did not do. Therefore, the guideline is strictly violated in the given LLM Response.\"\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "from uptrain import APIClient, Settings, ConversationGuidelineAdherence\n",
    "\n",
    "UPTRAIN_API_KEY = \"up-*******************************\"  # Insert your UpTrain API key here\n",
    "\n",
    "uptrain_client = APIClient(\n",
    "    Settings(\n",
    "        uptrain_access_token=UPTRAIN_API_KEY,\n",
    "    )\n",
    ")\n",
    "\n",
    "res = uptrain_client.log_and_evaluate(\n",
    "    \"Emergency-Contact-Information\",\n",
    "    data=data, \n",
    "    checks=[ConversationGuidelineAdherence(guideline=\"Provide emergency contact information if the patient is in distress\", guideline_name=\"Emergency Contact Information\")],\n",
    ")\n",
    "\n",
    "print(json.dumps(res, indent=3))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
